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VVhat  is  Technology? 


I.a.  The  application  of  science,  especially  to  industrial  or  commercial 
objectives*,  b.  The  entire  body  of  methods  and  materials  used  to 
achieve  such  objectives.” 

-  The  American  Heritage  Dictionary 


"or  in  NASA’s  case  space 
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Programs  &  Projects 


What  is  Technology  Assessment? 


•It  is  a  continuous,  iterative  process  that  must  begin  at  the  earliest 
stage  of  a  pre-program  (i.e.  concept  development  -  Pie-Phase  A)  and 
continues  throughout  the  life  of  the  program  until  final  design  and 
development  begins  (i.e.  Phase  C/D.) 

•It  in  fact  provides  critical  products  for  key  decision  points  (KDPs), 
(or  gates)  that  allow  transition  between  progranVproject  phases. 
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What  is  Technology  Assessment? 


It  is  a  two-step  process  that  involves: 


1.  The  accurate  determination  of  the  Technology  Readiness  Levels 
(TRLs)  (i.e.  current  level  of  maturity). 

2.  The  accurate  determination  of  the  Advancement  Degree  of 
Difficulty  (AD2)  (i.e.,  what  is  required  to  advance  a  technology 
from  its  current  TRL  to  what  is  required  for  infusion  into  the 
prograrrVproject  at  an  acceptable  level  of  risk.) 
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At  its  most  basic,  the  TRL  is  a  description  of  the  maturity  of  a  given 
technology  defined  by  what  has  been  done,  under  what  conditions  at 
a  given  point  in  time. 

•However,  the  TRL  is  just  one  part  of  the  equation  —  it  establishes  the 
baseline. 


•The  more  fundamental  question  is  what  is  required  (in  terms  of  cost, 
schedule  and  risk  to  move  the  technology  from  where  it  is  to  where  it 
needs  to  be. 


•In  addition,  there  is  an  organizational  aspect  of  technology 
assessment  that  speaks  to  the  capability  of  a  given  organization  to 
reproduce  a  technology  irrespective  of  its  maturity  level. 
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VN/hat  is  an  Advancement  Degree  of  Difficulty  (AD2)? 

AD2  is  a  method  of  dealing  with  the  other  aspects  beyond  TRL,  it  is 
the  description  of  what  is  required  to  move  a  technology  from  one 
TRL  to  another. 

•It  also  takes  into  account: 

— the  organizational  aspects  (ability  of  an  organization  to 
reproduce  existing  technology) 
manufacturability  (MRL) 

— Integration  (IRL) 

—Tools  &  facilities  (CRL) 

-  Human  Readiness  Levels  (HRL)  (skills) 

Capability  Readiness  Levels  (CRL)  (people  and  tods) 
—Software  Readiness  Levels  (SRL)  (but  not  wall) 
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1-  Culture: 

There  really  are  four  very  distinct  cultural  differences  among  the 
community  involved  in  any  typical  program 

•  Scientists,  who  know  all  there  is  to  know  about  science  and 
furthermore  think  they  know  everything  about  engineering  and 
technology. 

Engineers,  who  know  all  there  is  to  know  about  engineering,  think 
they  know  everything  about  technology  and  don’t  give  a  ******  about 
science. 
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•  Technologists,  who  really  do  know  all  there  is  to  know  about 
technology,  know  what  they  need  to  know  about  science  and  could  do 
engineering  if  they  wanted  to. 

•  Everyone  else  (including  Program  Managers) 

Now  the  main  thing  in  common  with  the  first  3  groups  is  that  they  a[l 
agree  that  the  4th  group  doesn’t  know  anything  about  anything  — 
especially  Program  Managers. 
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•The  Agency’s  programs  and  projects  by  their  very  nature 
require  the  development  and  infusion  of  technology  in 
order  to  meet  requirements. 


•Failure  to  do  this  successfully  is  one  of  the  major 
contributors  to  cost  and  schedule  overruns. 


•The  following  4  slides  are  excerpts  from  a  GAO 
presentation  on  their  assessment  of  the  reasons  for  cost 
and  schedule  overruns. 
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! 

GAO  Example  of  Requirements/Resource 
Mismatch  &  Resulting  Overrun 


Requirements  added 


Customer 
says  no 


I 


Match 

achieved 


Developer  reports 
197%  cost  increase 
&  15-month  delay  . 


tr- 

*  > 
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Knowledge  Building  for  Development 
of  New  Products  -  GAO  Study  Results 


- 


Technologies 

match 

requirements 


Design 
performs  as 
expected 


Production  can  meet 
cost,  schedule,  and 
quality  targets 


Commercial 
Knowledge  Points 


Program  launch 


DOD 

Knowledge  Points 
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Reconciling  Expectations,  Resources,  and 
Requirements  -  GAO  Study  Results 
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Timing  of  Systems  Engineering  - 
GAO  Study  Results 
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ProqrarrVproiect  Risk 


In  a  recent  article  in  the  Scan  Management  review,  uncertainty  (risk) 
was  broken  down  into  4  categories: 

1 .  Variation 

2.  Foreseen  uncertainty 

3.  Unforeseen  uncertainty  (unknown-unknowns) 

4.  Chaos 

These  four  categories  of  uncertainty  require  different  approaches 
from  the  management  team  if  they  are  to  be  successfully  resolved. 


* 


J 


•  Characterizing  the  Uncertainty  in  Projec 


Variation 


Characterizing  the  Uncertainty  in  Projects 


cjj 

Unforeseen  Uncertainty 


S3J 


Unforeseen  events  completely  invalidate  the  project’s  target,  planning  and  approach. 

•  The  project  team  must  continually  redefine  the  project’s  basic  premises  and  create  new  decision 

trees  based  on  incremental  learning. 


Medium-  and  long-term  contingencies  are  not  plannable. 


i* 
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Uncertainty  Profile 


•  Creating  an  uncertainty  profile  of  a  program  or  project  can 
provide  valuable  information  relative  to  what  is  required  to 
manage  a  program  successfully. 


•  VUiile  there  are  many  different  elements  that  contribute  to 
progranrVproject  uncertainty,  a  good  case  can  be  made  for  lack 
of  technology  maturity  being  the  dominant  component  in  the 
latter  two  categories: 

•Unforeseen  Uncertainty 
•Chaos. 
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Uncertainty  Profile 


•  Uncertainty  profiles  can  be  created  based  on  Technology 
Readiness  Levels  in  combination  with  their  associated 
Advancement  Degree  of  Difficulty. 


•  Different  Right  Programs  will  have  different  uncertainty  profiles 
depending  upon  the  amount  and  maturity  of  the  technologies 
that  must  be  infused  for  the  program  to  be  successful  and  the 
difficulty  required  in  advancing  the  technologies  to  the  point 
where  they  can  be  successfully  infused. 


8/13/2009 


21 


* 


Uncertainty  Profile 


Chaos 


Unforeseen 

Uncertainty 


Foreseen 

Uncertainty 


Variation 


Beware 


Technology  Assessment  Requirements  for 


Programs  &  Projects 


The  Assessment  Process 
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Technology  Assessment  Requirements  for 

Programs  &  Projects 


What  is  a  Technology  Readiness  Level  Assessment? 


•  It  is  the  assessment  of  the  state-of-the-art  (i.e.  maturity)  of  a  given 
technology  relative  to  the  categories  described  by  the  Technology 
Readiness  Levels. 

•  For  a  system,  subsystem  or  element,  the  TRL  for  the  whole  is 
determined  by  the  lowest  TRL  of  its  components. 

•At  its  most  basic  level,  the  TRL  is  a  description  of  what  has  been  done 
at  a  given  point  in  time. 


NB:  Operation  results  are  critical  to  determining  TRLs.  Tests  must  be 
done  in  the  proper  environment  and  the  unit  tested  must  be  of  an 
appropriate  scale  and  fidelity. 
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••  # 


Actual  system  “flight  proven”  through  successful  mission 
operations 

Actual  system  completed  and  “flight  qualified”  through  test 
and  demonstration  (Ground  or  Flight) 

System  prototype  demonstration  in  a  space  environment 

System/subsystem  model  or  prototype  demonstration  in  a 
relevant  environment  (Ground  or  Space) 

Component  and/or  breadboard  validation  in  relevant 
environment 

Component  and/or  breadboard  validation  in  laboratory 
environment 

Analytical  and  experimental  critical  function  and/or 
characteristic  proof-of-concept 

Technology  concept  and/or  application  formulated 
Basic  principles  observed  and  reported 
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What  is  a  Technology  Readiness  Level  Assessment? 


The  TRL  Assessment  starts  with  a  definition  of  the  terms  used  in  the 
TRL  descriptions,  without  a  common  set  of  definitions,  the 
assessment  will  be  of  marginal  use! 
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Prototype  Unit:  The  prototype  unit  demonstrates  form,  fit  and 
function.  It  is  to  every  possible  extent  identical  to  flight  hardware, 
and  is  built  to  test  the  manufacturing  and  testing  processes  and 
is  intended  to  be  tested  to  flight  qualification  levels,  the  only 
difference  from  the  flight  unit  is  that  it  is  realized  that  elements  of 
the  prototype  unit  will  in  all  probability  be  changed  as  a  result  of 
experiences  encountered  in  the  development  and  testing  of  the 
Prototype  unit. 


Relevant  Environment:  Not  all  systems,  subsystems  and/or 
components  need  to  be  operated  in  a  full  space/launch 
environment  in  order  to  satisfactorily  address  performance 
margin  requirements.  Consequently,  the  specific  environment  is 
tailored  to  the  performance  requirements  being  addressed. 
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Technology  Assessment  Requirements  for 

Programs  &  Projects  *  * 


Technology  Assessment  Requirements  for 

Programs  &  Projects 


Once  definitions  have  been  agreed  to  then  a  series 
of  questions  may  be  asked  that  will  determine  the 

TRL 
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Wr  Programs  &  Projects 


W:  ••  c 


NO 
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Programs  &  Projects 


••  i 


NO 
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Beware  -  Land  of  the  Unknown 
(There  be  monsters  here) 


31 


•  Repeat  the  process  for  all  subsystems,  identifying  the  TRLs 
corresponding  to  each  subsystem 

•  Repeat  the  process  for  all  elements  of  each 

subsystem,  identifying  the  TRL  corresponding  to  each  element 
within  a  subsystem 

The  lowest  TRL  of  the  lowest  element  is  the  TRL  of  the 
system 


NB:  Although  TRL  6  is  often  referred  to  as  the  “magic”  level  above 
which  there  is  no  problem,  in  fact  the  AD2  assessment  must  be  done  in 
order  to  quantify  the  cost,  schedule  and  risk  even  for  those 
technologies  above  TRL  6. 
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Appropriate  Scale 


Technology  Assessment  Requirements  for 
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VMiat  is  an  Advancement  Degree  of  Difficulty  (AD2)? 


•Identifying  the  TRLs  of  the  systems,  subsystems  and  components  is 
just  one  part  of  the  equation  —  it  establishes  the  baseline  maturity. 

•The  more  fundamental  question  is  what  is  required  (in  terms  of  cost, 
schedule  and  risk  to  to  advance  the  maturity  to  a  level  necessary  for 
successful  incorporation  into  the  program 


•The  AD2  incorporates: 

manufacturability  (Manufacturing  Readiness  Levels  -  MRLs) 
”  integration  (Integration  Readiness  Levels  -  IRLs) 

tools  and  capabilities  (Capability  Readiness  Levels  -  CRLs) 
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Specifically  with  respect  to  the  CRL,  there  is  an  organizational  aspect  of 
technology  assessment  that  speaks  to  the  capability  of  a  given 
organization  to  reproduce  a  technology  irrespective  of  its  maturity  level. 
Many  programs  have  encountered  major  difficulties  with  “heritage” 
technology  that  either  could  not  be  reproduced. 

Additionally  the  use  of  “heritage”  systems  and  components  often  results 
in  important  systems  engineering  steps  being  omitted  under  the  belief 
that  they  have  already  been  addressed  —  this  often  leads  to  disastrous 
consequences. 
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Overall,  AD2  assessment  is  one  of  the  most  challenging  aspects  of 
technology  assessment.  —  the  cost ,  schedule  and  risk  associated  with 
advancing  maturity  levels  varies  greatly  with  the  technology. 

Appropriately  assessing  A requires  the  art  of  “prediction,”  which,  if  it  is 
to  be  accurate  must  rely  on: 

•  Expert  personnel 

Detailed  examination  of  required  activity. 

•  Review  by  independent  advisory  panel 
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Having  acquired  the  appropriate  expertise,  determination  of  the 
AC£  is  primarily  a  matter  of: 

Addressing  the  appropriate  questions  regarding  the  development 
process 

•  Identifying  the  quantitative  steps  in  the  developments  that  must  be 
undertaken  (breadboards,  developmental  models,  prototypes,  etc.) 

•  Identifying  what  tests  must  be  undertaken  to  certify  the  advancement 

•  Making  informed  assessments  of  the  degree  of  difficulty  in  pursuing 
the  development/testing/evaluation. 
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Technology  Assessment  Requirements  for 

Programs  &  Projects  :i§f 

AD2  Assessment  —  detailed  examination  of  required 

activity 


Design/Anal  vsis: 

Do  you  have  the  necessary  tods  for  design  and  analysis  at  the 
level  of  accuracy  required?  If  not  what  needs  to  be  done,  how  long 
will  it  take  and  how  difficult  will  it  be  to  accomplish  it? 

•  Data  bases 

•  Design  methods 

•  Analytical  tods 

•  Models 
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Technology  Assessment  Requirements  for 

Programs  &  Projects 

AD2  Assessment  —  detailed  examination  of  required 

activity 


Manufacturing: 

Do  you  have  the  necessary  tools/processes  for  manufacturing  at 
the  level  of  accuracy  required?  If  not  what  needs  to  be  done,  how 
long  will  it  take  and  hew  difficult  will  it  be  to  accomplish  it? 


Materials 

Metrology  Process 
development 


Tooling 

Developmental  units 
required 
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Technology  Assessment  Requirements  for 

Programs  &  Projects  :i§f 

AD2  Assessment  —  detailed  examination  of  required 


activity 


Test  &  Evaluation: 


Do  you  have  the  necessary  equipment/processes/facilities  for  test 
and  evaluation  at  the  level  of  accuracy  required?  If  not  what  needs  to 
be  done,  how  long  will  it  take  and  how  difficult  will  it  be  to  accomplish 
it? 


Environmental  Facilities 

Analysis  Software 

Test  units  needed  (breadboards, 
prototypes  etc.) 


Test  Hardware 


Special  requirements 
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Technology  Assessment  Requirements  for 

Programs  &  Projects  :i§f 

AD2  Assessment  —  detailed  examination  of  required 

activity 


Operability: 

Throughout  the  development  of  the  design,  manufacturing  and 
testing  processes,  operability  must  be  taken  into  account. 


•  Ease  of  manufacture  •  Operability  •  Reproducibility 


•  Reliability 

•  Life  cycle  costs 


•  Verifiability  •  Testability 

•  Sustainability 
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Technology  Assessment  Requirements  for 

’•  C 

* 

AD2  Assessment  Matrix 

The  answers  to  the  questions  can  again  be  formulated  into  a  matrix 
that  indicates  the  level  of  risk  associated  with  each  area  and  provides 
the  basis  for  subsequent  cost  and  schedule  estimates. 


Programs  &  Projects 


Not  applicable 

Design  &  Analysis 

Manufacturing 

Operability 

Test  &  Evaluation 

Demonstration  Units 

Overall  Numerical  Score  | 

E 
c r ) 

C f ) 

0 
< n 

< 

0 

> 

"co 

O 

"ro 

0 

> 

O 

Red  =  Major  Concern 

Yellow  =  Moderate  Concern 

Green  =  OK 

White  =  Under  Evaluation 

Existing  =  1.0 

Moderate  Risk  =  5.0 

High  Risk  =  8.0 

Major  Breakthrough  =  10.0 

Databases 

Design  Methods 

Analytical  Tools 

Models 

Components 

Materials 

Facilities 

Machines 

Tooling 

Metrology 

Mfg.  Software 

Mfg.  Processes 

Assembly  &  Alignment 

Life  Cycle  Costs 

Reproducibility 

co 

3 

o 

n 

c 

CO 

Process  Variability 

Verifability 

Testability 

Reliability 

Integration  Issues 

Special  Facilities 

Environmental  Facilities 

Special  Software 

Special  Test  Equipment 

Special  Support  Equipmen 

Breadboard 

Brassboard 

Scale  Model 

Engineering  Unit 

Prototype 

1 .0  System 

1.1  Subsystem 

1.1.1  Components 

1.1.2 

1.1.3 

1.1.4 

1.1.5 

1.1.6 

1.1.7 

1.1.8 

1.1.9 

1.1.10 

1.1.11 

1.1.12 

1 .2  Subsystem 

1.2.1  Components 
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Technology  Assessment  Acquirements  for 

Programs  &  Projects 

•  |  ■  ;  ; 

Key  Decision  Points 

The  information  contained  in  the  TRL  matrix  and  the  corresponding 
AC£  matrix  provides  the  information  required  for  key  decision  points 
(KDPS) 

^  Identifies  Critical  Technologies 

Provides  for  Information  for  Risk  IVStigation  Ran  Preparation 
Breadboards  and  Developmental  Models  Required 
—Tests  Required 
—Alternate  Approaches 

—Fall  Back  Positions  and  Corresponding  Performance 
Reductions 
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Technology  Assessment  Requirements  for 

Programs  &  Projects 

■  - .  •  !  |  ■  ; 

Cost  and  Schedule 


The  AC£  assessment  provides  considerable  detail  for  an  accurate 
determination  of  program  cost  and  schedule. 

The  identification  of  data  bases,  tods,  processes,  facilities  tests, 
scale  model  development  and  integration  issues  in  particular  will 
assist  in  developing  realistic  cost  plans. 

The  identification  of  requirements  for  engineering  model 
development  and  subsequent  tests  will  be  of  particular  benefit  in 
outlining  realistic  schedules. 
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PrograrrVPrqject/Activity  Technology  Assessment  Process 
Summary 


; 


>  Q early  define  all  terminology  used  in  the  Technology 
Readiness  Level  (TRL)  descriptions  to  be  used  throughout  the 
life  of  the  prograrr/project/activity. 

>  Provide  a  formal  assessment  of  the  TRL  for  each  system, 
subsystem  and  component  as  described  by  the 
prograrrYproject  V\BS.  (The  “weakest  link’  concept  will  be 
used  in  determining  system  and  subsystem  TRLs  wherein  the 
TRL  of  the  system  is  determined  by  the  subsystem  having  the 
lowest  TRL  in  the  system  which  in  turn  is  determined  by  the 
component  having  the  lowest  TRL  in  the  subsystem 
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Technology  Assessment  Requirements  fir 

Programs  &  Projects  ^ 

*  '  '  , 

PrograrrVPrqject/Activity  Technology  Assessment  Requirements 

>  On  the  basis  of  the  assessment  prepare  a  Critical  Technology  List, 

i.e.,  a  list  of  those  technologies  that  are  absolutely  essential  in 
meeting  requirements  and  that  have  substantial  risk,  cost,  and/or 
schedule  involved  in  the  development 


>  Prepare  a  risk  mitigation  plan  (Advancement  Degree  of  Difficulty 
(AD2)  for  each  critical  technology  that  addresses  the  cost,  schedule 
and  risk  associated  with  advancing  each  element  to  the  point 
necessary  to  meet  requirements  in  a  timely  manner.  Identify 
alternative  paths,  decision  gates,  off-ramps,  fallback  positions,  and 
quantifiable  milestones  with  appropriate  schedules  (Technical 
Performance  Metrics  (TPMs)  that  measure  progress  towards 
requirements  and  are  verified  by  test. 
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*  .4-  *  1 

PrograrrVPrqject/Activity  Technology  Assessment  Requirements 


>  Progress  on  risk  mitigation  will  be  tracked  at  periodic  reviews  until 
COR  when  all  appropriate  risk  is  deemed  to  have  been  eliminated  or 
reduced  to  acceptable  levels. 


8/13/2009 


47 


Technology  Assessment  Requirements  for 

Programs  &  Projects 


Summary 


Successful  development  and  incorporation  of  technology 
into  programs  is  a  hard  job!  But  —  it  is  not  magic,  it  simply 
requires  up  front  application  of  systems  engineering. 

It  takes  time  and  money  and  effort,  but  in  the  end,  it  must 
be  done  if  the  prograrr/project  is  to  sucoeed. 
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